Organise output csv by Mariia-Var · Pull Request #508 · simpaths/SimPaths

Mariia-Var · 2026-08-24T14:01:20Z

No description provided.

-Drop the entity-id suffix from output filenames (Statistics21.csv -> DemographicStatistics.csv) by exporting singleton collections rather than bare objects. Row values are unchanged. -Split Statistics2, which held 51 columns across four domains, and rename the outputs for what they contain: Statistics -> WealthIncomeStatistics Statistics2 -> DemographicStatistics EmploymentStatistics -> LabourStatistics AlignmentAdjustmentFactors -> AlignmentStatistics HealthStatistics -> WellbeingByGender -HealthStatistics is now a new output for population health by age band. AgeBandAggregates computes the shared age-band statistics once per year, so the independently-toggled outputs cannot drift apart. -Remove 12 calibration residuals reporting simulated minus an outdated 2019 UKHLS target; nothing read them. -Add OutputReadme, writing README.md beside the CSVs with the run configuration (population size, years, runs, alignment, bootstrapAll). -Config keys follow the field renames; old keys are silently ignored.

Copilot

Pull request overview

This PR reorganises SimPaths’ annual CSV outputs by splitting the previous monolithic statistics exports into clearer, topic-specific files (wealth/income, demographics, labour, health, wellbeing, alignment), updating persistence configuration, integration tests, and documentation accordingly, and adding an output-folder README generator to preserve run configuration context alongside results.

Changes:

Replaces legacy Statistics* exports with new WealthIncomeStatistics, DemographicStatistics, LabourStatistics, HealthStatistics, WellbeingByGender, and AlignmentStatistics outputs.
Updates SimPathsCollector scheduling/export flow (including shared AgeBandAggregates) and adjusts model/caller code to use the new statistics objects.
Refreshes integration tests + expected CSV fixtures and updates repository documentation/configs to reflect the new filenames and semantics.

Reviewed changes

Copilot reviewed 32 out of 32 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/test/java/simpaths/integrationtest/RunSimPathsIntegrationTest.java	Updates integration assertions/comparisons to the new CSV filenames and adds coverage for new outputs.
src/test/java/simpaths/integrationtest/expected/WellbeingByGender.csv	New expected output fixture for wellbeing-by-gender export.
src/test/java/simpaths/integrationtest/expected/WealthIncomeStatistics.csv	New expected output fixture replacing legacy `Statistics1.csv`.
src/test/java/simpaths/integrationtest/expected/Statistics21.csv	Removes legacy expected fixture superseded by new split outputs.
src/test/java/simpaths/integrationtest/expected/Statistics1.csv	Removes legacy expected fixture superseded by `WealthIncomeStatistics.csv`.
src/test/java/simpaths/integrationtest/expected/LabourStatistics.csv	New expected output fixture for labour statistics export.
src/test/java/simpaths/integrationtest/expected/HealthStatistics.csv	New expected output fixture for age-band health export.
src/test/java/simpaths/integrationtest/expected/EmploymentStatistics1.csv	Removes legacy expected fixture superseded by `LabourStatistics.csv`.
src/test/java/simpaths/integrationtest/expected/DemographicStatistics.csv	New expected output fixture for demographic age-band export.
src/test/java/simpaths/integrationtest/expected/AlignmentStatistics.csv	Updates expected fixture to the renamed alignment output.
src/test/java/simpaths/data/statistics/LabourStatisticsTest.java	Renames the test class/display name to match the new `LabourStatistics` naming.
src/main/resources/META-INF/persistence.xml	Updates JPA class list to new statistics entities and removes legacy ones.
src/main/java/simpaths/model/SimPathsModel.java	Adds getters needed for README/export reporting of model options.
src/main/java/simpaths/model/LabourMarket.java	Switches percentile/quintile lookups to the new wealth/income statistics object.
src/main/java/simpaths/model/BenefitUnit.java	Switches income-quintile updates to the new wealth/income statistics object.
src/main/java/simpaths/experiment/SimPathsMultiRun.java	Adds explicit multirun-mode tracking/accessors used by output README reporting.
src/main/java/simpaths/experiment/SimPathsCollector.java	Refactors collection/export pipeline to new stats objects, adds shared `AgeBandAggregates`, and writes an output README when exporting CSVs.
src/main/java/simpaths/experiment/OutputReadme.java	New utility to write `README.md` beside CSV outputs describing run configuration + files.
src/main/java/simpaths/data/statistics/WellbeingByGender.java	New entity producing stacked wellbeing/QoL outputs by gender for ages 25–64.
src/main/java/simpaths/data/statistics/WealthIncomeStatistics.java	New entity holding wealth/income distribution and age-band aggregates.
src/main/java/simpaths/data/statistics/Statistics2.java	Removes legacy monolithic statistics entity superseded by split outputs.
src/main/java/simpaths/data/statistics/Statistics.java	Removes legacy monolithic statistics entity superseded by split outputs.
src/main/java/simpaths/data/statistics/LabourStatistics.java	Renames/extends labour statistics entity and adds age-band FT/PT shares (via aggregates).
src/main/java/simpaths/data/statistics/HealthStatistics.java	Refactors to age-band health/disability outputs (wellbeing moved to `WellbeingByGender`).
src/main/java/simpaths/data/statistics/DemographicStatistics.java	New entity for demographics by age band.
src/main/java/simpaths/data/statistics/AlignmentStatistics.java	Renames alignment adjustment factors entity to align with new output naming.
src/main/java/simpaths/data/statistics/AgeBandAggregates.java	New shared computation of age-band aggregates used by multiple annual outputs.
documentation/wiki/developer-guide/repository-guide.md	Updates developer documentation to list the new output CSV filenames and meanings.
documentation/repository-guide.md	Updates repository guide output-file listing to match new structure.
config/test_run.yml	Updates integration test config keys for renamed collector persistence toggles.
config/default.yml	Updates collector output documentation/comments to reflect new outputs.
CLAUDE.md	Updates architecture/output documentation to reflect new output filenames and shared aggregate approach.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

  # The S-Index is an economic (in)security index computed from a rolling window of
  # equivalised consumption, discounted and weighted by a risk-aversion parameter.
-  # SIndex_p50 is reported in Statistics1.csv each year.
+  # SIndex_p50 is reported in Statistics.csv each year.


+        for (int ii=0; ii<=2; ii++) {
+
+            if (popula[ii]>=0) {
+
+                labInc[ii] /= (workFT[ii] + workPT[ii]);
+                prMarr[ii] /= popula[ii];
+                avkids[ii] /= popula[ii];
+                health[ii] /= popula[ii];
+                prDisa[ii] /= popula[ii];
+                workFT[ii] /= popula[ii];
+                workPT[ii] /= popula[ii];
+                invInc[ii] /= popula[ii];
+                penInc[ii] /= popula[ii];
+                invLosses[ii] /= popula[ii];
+                grossDisInc[ii] /= popula[ii];
+                wealth[ii] /= popula[ii];
+            }


+    public WealthIncomeStatistics getWealthIncomeStats() {
+        return wealthIncomeStats;
    }

-    public void setStats(Statistics stats) {
-        this.stats = stats;
+    public void setStats(WealthIncomeStatistics wealthIncomeStats) {
+        this.wealthIncomeStats = wealthIncomeStats;
    }

-    public Statistics2 getStats2() { return stats2; }
+    public DemographicStatistics getStats2() { return demographicStats; }

-    public void setStats2(Statistics2 stats2) { this.stats2 = stats2; }
+    public void setStats2(DemographicStatistics demographicStats) { this.demographicStats = demographicStats; }


Copilot

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

src/main/java/simpaths/data/statistics/AgeBandAggregates.java:104

In the age-band normalisation loop, if (popula[ii] >= 0) is always true (population starts at 0), so this block will divide by zero when an age band is empty. Also, labInc[ii] /= (workFT[ii] + workPT[ii]) divides by zero when there are no workers in the band. Guard both denominators so empty bands / zero-worker bands don’t silently produce NaN/Infinity.

        for (int ii=0; ii<=2; ii++) {

            if (popula[ii]>=0) {

                labInc[ii] /= (workFT[ii] + workPT[ii]);

src/main/java/simpaths/experiment/SimPathsCollector.java:883

The setter name setStats(...) no longer matches the field/getter (wealthIncomeStats / getWealthIncomeStats()), which makes the public API confusing (and makes reflective tooling harder to follow). Consider keeping setStats as a deprecated alias and adding a correctly named setWealthIncomeStats method.

    public void setStats(WealthIncomeStatistics wealthIncomeStats) {
        this.wealthIncomeStats = wealthIncomeStats;
    }

src/main/java/simpaths/experiment/SimPathsCollector.java:887

Similarly, getStats2() / setStats2(...) now return/accept DemographicStatistics, so the method names are misleading. Adding clearly named accessors (and leaving the old names as deprecated aliases) will reduce confusion and make future refactors safer.

    public DemographicStatistics getStats2() { return demographicStats; }

    public void setStats2(DemographicStatistics demographicStats) { this.demographicStats = demographicStats; }

config/default.yml:46

This comment still references Statistics.csv, but the S-Index is now written to WealthIncomeStatistics.csv. Updating this avoids misleading configuration documentation.

  # SIndex_p50 is reported in Statistics.csv each year.

+    public void setGender(String demSex) {
+        this.demSex = demSex;
+    }


- Guard both denominators in AgeBandAggregates: empty age bands and bands with no workers no longer divide by zero. Non-empty bands are unchanged. - Give WellbeingByGender a distinct entity id per gender group (Total=1, Male=2, Female=3); the constant id collided on the JPA primary key across the three rows written each year. - Rename setStats/getStats2/setStats2 to match their fields. - Fix stale Statistics.csv reference in default.yml.

Copilot

Pull request overview

Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.

…ender.java Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Mariia-Var added 2 commits August 24, 2026 14:51

Copilot AI lite review requested due to automatic review settings August 24, 2026 14:01

Copilot started reviewing on behalf of Mariia-Var August 24, 2026 14:01 View session

Copilot AI reviewed Aug 24, 2026

View reviewed changes

Mariia-Var changed the title ~~Mv organise output csv~~ Organise output csv Aug 24, 2026

updated default.yml

11c244a

Mariia-Var requested a lite review from Copilot August 24, 2026 14:57

Copilot started reviewing on behalf of Mariia-Var August 24, 2026 14:57 View session

Copilot AI reviewed Aug 24, 2026

View reviewed changes

Comment thread src/main/java/simpaths/data/statistics/WellbeingByGender.java

Comment on lines +124 to +126

public void setGender(String demSex) {

this.demSex = demSex;

}

Mariia-Var requested a lite review from Copilot August 24, 2026 15:15

Copilot started reviewing on behalf of Mariia-Var August 24, 2026 15:16 View session

Copilot AI reviewed Aug 24, 2026

View reviewed changes

Comment thread src/main/java/simpaths/data/statistics/WellbeingByGender.java

removed unused import jakarta.persistence.Transient from WellbeingByG…

e96bc51

…ender.java Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

dav-sonn merged commit bef7f26 into simpaths:develop Aug 25, 2026
6 checks passed

Mariia-Var deleted the MV-organise-output-csv branch August 28, 2026 15:21

Navigation Menu

Uh oh!

Organise output csv - #508

Organise output csv#508
dav-sonn merged 5 commits into
simpaths:developfrom
Mariia-Var:MV-organise-output-csv

Mariia-Var commented Aug 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Mariia-Var commented Aug 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants